Transmission of 3D models

ABSTRACT

A method and an apparatus for transmitting a 3D model associated to stereoscopic content are described, and more specifically a method and an apparatus for the progressive transmission of 3D models. Also described are a method and an apparatus for preparing a 3D model associated to a 3D video frame for transmission and a non-transient recording medium comprising such a prepared 3D model. The 3D model is split into one or more components. It is then determined whether a component of the one or more components is hidden by other 3D content. For transmission those components of the 3D model that are not hidden by other 3D content are transmitted first. The remaining components of the 3D model are transmitted subsequently.

FIELD OF THE INVENTION

The present invention relates to a method and an apparatus fortransmitting a 3D model associated to stereoscopic content, and morespecifically to a method and an apparatus for the progressivetransmission of 3D models. The invention further relates to a method andan apparatus for preparing a 3D model associated to a 3D video frame fortransmission, and to a recording medium including such a prepared 3Dmodel.

BACKGROUND OF THE INVENTION

3D models are typically comprised of a set of connected points in 3Dspace, commonly referred to as vertices. The points are connected insuch a way that a polygonal mesh structure is formed. In the mostgeneral case, the polygons may have any number of sides. In thefollowing it is assumed that the polygons are all triangles. However, asany planar polygon can be divided into a set of triangles, the describedgraphics process does not suffer any loss of generality.

The introduction of 3D visualization is expected to be the nexttechnological differentiator in consumer-oriented video devices. In thepast few years, the complete chain, from the content generation to theviewing experience, has become increasingly mature. Many standardorganizations have launched study groups on the transmission of 3Dcontent. More and more 3D distributed applications increase the demandfor efficient transmission of 3D models. Some distributed virtualreality (VR) applications even demand real-time on-request transmissionof 3D models. Generally a standard client-server architecture isemployed, in which a central server maintains a geometry database of thevirtual environment and distributes object models to clients uponrequest. Because these 3D models may be complex and are usually large innumber, the network bandwidth often becomes the bottleneck of such asystem.

Currently there are mainly two approaches to encode the 3D models toreduce the amount of information that has to be sent through thenetwork. The first approach is to apply a geometry-compression method toreduce the storage size of the models. Most geometry-compression methodsconsider the geometry information shared by neighboring polygons andreduce the amount of data needed to represent the polygon mesh.

The second approach is to encode the 3D models for progressivetransmission. For this approach the models are converted in such a waythat partially transmitted models can be rendered and progressivelyrefined as more information is received. As a result, the client nolonger needs to wait for the whole model to be transmitted beforerendering, and it can thus provide a more immediate visual feedback tothe user.

For the latter approach, the 3D model is decomposed into a base mesh anda sequence of progressive records. The base mesh represents theminimum-resolution model of the 3D object. A progressive record storesinformation of a vertex split that may slightly increase the resolutionof the base mesh by introducing two triangles into it. Hence, byapplying the sequence of progressive records to the base mesh, the modelwill gradually increase in resolution until it reaches the highestresolution when all the records have been applied. The resolution of themodel can be decreased by reversing the above operation.

A few progressive methods have been developed in the past years.However, all these methods are based on the principle that the partiallytransmitted model should look like the original complete model with asubsequently increasing resolution. Therefore, initially the viewers arealways confronted with a low resolution model.

More and more 3D distributed applications increase the demand forefficient transmission of 3D models. For example, in a 3D TVtransmission network such as MPEG TS (MPEG Transport Stream), DVB(Digital Video Broadcasting) and CMMB (China Multimedia MobileBroadcasting), many 3D programs are transmitted concurrently. 3D modelsneed to be transmitted and rendered along with relevant 3D videos.Usually a 3D model has to be received completely before it can berendered by a client. As these 3D models may be complex and are usuallylarge in number, the network bandwidth often becomes the bottleneck ofthe transmission system.

SUMMARY OF THE INVENTION

It is an object of the invention to propose a method and a system fortransmitting 3D models, which allow to operate with reduced requirementsfor the network bandwidth.

According to the invention, this object is achieved by a method fortransmitting a 3D model associated to a 3D video frame, having the stepsof:

-   -   splitting the 3D model into one or more components;    -   determining whether a component of the one or more components is        hidden by other 3D content;    -   transmitting those components of the 3D model that are not        hidden by other 3D content; and    -   subsequently transmitting the remaining components of the 3D        model.

Similarly, a method for preparing a 3D model associated to a 3D videoframe for transmission has the steps of:

-   -   splitting the 3D model into one or more components; and    -   determining whether a component of the one or more components is        hidden by other 3D content.

Before transmission the 3D models that are going to be transmitted aresplit into different components. When the server device begins totransmit the 3D models, the components that are not hidden by other 3Dcontent, e.g. 3D video, other components of the 3D model, other 3Dmodels, and so on, are transmitted first. As a result, it is possiblethat only a part of a 3D model has been transmitted when the clienttries to render the 3D model. However, since those components that arenot transmitted are hidden by the content of 3D video or other 3Dmodels, the viewer will not notice that the 3D model is incomplete. Asthe transmission progresses, the remaining components of the 3D modelsare transmitted to the client by the server device. Therefore, theclient will finally receive complete 3D models. The proposed solutionhas the advantage that only a reduced bandwidth is necessary to transmitthe 3D models, while at any time the viewer experiences a highestresolution of the models.

Preferably, a list of components of the 3D model is generated, in whichthe components of the 3D model are ordered in accordance with theirdepth. This allows to easily determine at least which components arehidden by other components of the 3D model. By generating a single listof components for all 3D models associated to the 3D video frame, inwhich the components of all 3D models are ordered in accordance withtheir depth.

In order to implement the invention, an apparatus for transmitting a 3Dmodel associated to a 3D video frame has:

-   -   a first analyzer for splitting the 3D model into one or more        components;    -   a second analyzer for determining whether a component of the one        or more components is hidden by other 3D content; and    -   an output for transmitting those components of the 3D model that        are not hidden by other 3D content, and for subsequently        transmitting the remaining components of the 3D model.

Similarly, an apparatus for preparing a 3D model associated to a 3Dvideo frame for transmission has:

-   -   a first analyzer for splitting the 3D model into one or more        components; and    -   a second analyzer for determining whether a component of the one        or more components is hidden by other 3D content.

Advantageously, a list generator is provided, which generates a list ofcomponents of the 3D model, in which the components of the 3D model areordered in accordance with their depth. Preferably, the list generator 4is adapted to generate a single list of components for all 3D modelsassociated to the 3D video frame, in which the components of all 3Dmodels are ordered in accordance with their depth.

According to a further aspect of the invention, a recording mediumhaving recorded thereon 3D content, which includes at least one or more3D video frames and one or more 3D models associated to the one or more3D video frames, contains at least one list of components, in whichcomponents of the one or more 3D models associated to a 3D video frameare ordered in accordance with their depth.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding the invention shall now be explained in moredetail in the following description with reference to the figures. It isunderstood that the invention is not limited to this exemplaryembodiment and that specified features can also expediently be combinedand/or modified without departing from the scope of the presentinvention as defined in the appended claims. In the figures:

FIG. 1 shows a complete chain of stereoscopic content,

FIG. 2 depicts a cube model,

FIG. 3 shows the result of an incomplete model,

FIG. 4 illustrates a flow chart of a component ordering step,

FIG. 5 depicts a flow chart of a progressive transmission of componentsof models according to the invention; and

FIG. 6 schematically illustrates an apparatus adapted to implement theinvention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 shows an example complete chain of stereoscopic content, from thecontent generation to the display. As shown in the figure, apart fromthe video and audio streams, which consist of traditional 2D program, adepth stream is transmitted so that viewers can enjoy stereoscopicimages. Furthermore, a 3D models stream is advantageously transmitted toenhance the interactivity and 3D effects. All these streams aresynchronized.

A 3D model is a data structure that consists, for example, of a list ofvertices, a list of edges, a list of wedges, a list of faces and a listof material properties. A vertex represents a point in the 3D space,which can be represented by an ordered triplet (x, y, z). The firstentry of the triplet conventionally represents the horizontal positionand is typically denoted by x. The second entry represents the verticalposition and is typically denoted by y. Finally, the third numberrepresents the depth and is typically denoted by z.

An edge is a line joining two vertices. A wedge is an intermediateabstraction layer between vertices and faces, which is used toencapsulate the scalar attributes, such as normal vectors and texturecoordinates, of a vertex. A face is a triangular plane with exactlythree wedges at its corners. Material attributes represent the discreteattributes, such as ambient, diffuse, emissive, specular and shininess,of the faces, which are used to specify the appearance of the facesduring rendering.

FIG. 2 shows the geometry effect of a cube model. In this 3D model,there are eight vertices. Unlike video frames, which are just renderedfor an instant and then removed from the buffer, 3D models are stored inthe system and may be rendered for a long time. Usually a 3D model needsto be received completely before it can be rendered by a client.However, if some vertices are lost, the 3D model can still be rendered,but the result may not what it is expected. FIG. 3 is the renderingresult of the 3D model of FIG. 2 if the two vertices a and b of the topleft front and rear corner of the cube model are lost.

Each 3D model as considered in the present application is assumed tocomprise one or more components. A component is defined as an atomicpart of a model, which is semantically meaningful and can beindividually transmitted to a client for rendering purposes. Forexample, a small component can be comprised of a single face.

In a 3D transmission system, e.g. in a 3D TV broadcasting system, both3D video and 3D models are transmitted. In many cases some 3D models orsome components of a model will be hidden by the content of the 3Dvideo. The present invention makes use of this circumstance byprogressively transmitting the components of a model according to thedepth of the components and the video. Before transmitting a 3D model,the service device orders the components of the model by their depth sothat those components that are not hidden are transmitted first. Forthis purpose the depth of a component is defined as the depth of thefarthest vertical plane perpendicular to the z-axis that is able tocompletely hide the component. In other words, the depth of a componentis the minimum z-value of all vertices of the component, provided thatthe z-value increases with increasing distance of a component.

FIG. 4 depicts a flow chart of a component ordering step for one model.After selecting a component from the model the depth of the component iscalculated. The selected component is then inserted into an ordered listof components in accordance with the calculated depth. Subsequently itis checked whether all components of the model have been processed. Ifthis is not the case, the next component is selected and processed. Theprocedure is repeated until all components have been processed.

FIG. 5 is a flow chart showing how to progressively transmit componentsof models in a 3D transmission system in which both 3D video and 3Dmodels are transmitted. The server firstly orders the components of all3D models that are going to be transmitted in a single ordered list ofcomponents. Then the server gets the depth map of the video frame thatthese models are synchronized with. Next, the server runs through theordered list of components. For each component, the server checks if thecomponent is completely hidden by the video or by previously selectedcomponents. Only if the component is not completely hidden it isselected for transmission and removed from the ordered list ofcomponents. Since the list of components is ordered by the depth, acomponent later in the list will never completely hide a componentearlier in the list. As a result, all selected components are reliablynot completely hidden. Only these selected components are transmittedinitially.

When the server is going to transmit next video frame, it will look forall new 3D models synchronized with this new video frame. The componentsof these new models will be ordered and merged with the ordered list ofcomponents that were not transmitted along with previous video frame.Then the server will select components from the new ordered list ofcomponents that are not completely hidden by the video frame or otherselected components. Only these selected components are transmitted withthe video frame. This procedure goes on until all components of all 3Dmodels have been transmitted.

It is possible that only a part of a 3D model has been received when theclient tries to render it. However, since those components that are nottransmitted are hidden by the content of 3D video or by other 3D models,viewers will not notice that the 3D model is incomplete. With continuingtransmission the rest of the 3D model will be progressively transmittedto the client by the server device so that the client will finallyreceive a complete 3D model.

An apparatus 1 that is adapted to implement the present invention isschematically illustrated in FIG. 6. The apparatus 1 includes a firstanalyzer 2 for splitting a 3D model, which is associated to a 3D videoframe, into one or more components.

A second analyzer 3 determines whether a component of the one or morecomponents is hidden by other 3D content. For this purpose a listgenerator 4 generates a list of components of the 3D model, in which thecomponents of the 3D model are ordered in accordance with their depth.

Advantageously, the list generator 4 is adapted to generate a singlelist of components for all 3D models associated to the 3D video frame,in which the components of all 3D models are ordered in accordance withtheir depth. Of course, it is likewise possible that the second analyzer3 analyzes the components without the help of the ordered list ofcomponents, which is indicated by the dashed arrow in FIG. 6. The outputof the second analyzer 3 is a list of selected components that are to betransmitted first as well as list of components that are to betransmitted subsequently. The apparatus further has an output 5 fortransmitting the components of the 3D model as determined by the secondanalyzer 3.

1. A method for transmitting a 3D model associated to a 3D video frame,the method comprising the steps of: splitting the 3D model into one ormore components; determining whether a component of the one or morecomponents is hidden by other 3D content; transmitting those componentsof the 3D model that are not hidden by other 3D content; andsubsequently transmitting the remaining components of the 3D model. 2.The method according to claim 1, wherein the other 3D content is atleast one of 3D video, another component of the 3D model, and another 3Dmodel.
 3. The method according to claim 1, further comprising the stepof generating a list of components of the 3D model, in which thecomponents of the 3D model are ordered in accordance with their depth.4. The method according to claim 3, wherein a single list of componentsis generated for all 3D models associated to the 3D video frame, inwhich the components of all 3D models are ordered in accordance withtheir depth.
 5. A method for preparing a 3D model associated to a 3Dvideo frame for transmission, the method comprising the steps of:splitting the 3D model into one or more components; and determiningwhether a component of the one or more components is hidden by other 3Dcontent.
 6. The method according to claim 5, wherein the other 3Dcontent is at least one of 3D video, another component of the 3D model,and another 3D model.
 7. The method according to claim 5, furthercomprising the step of generating a list of components of the 3D model,in which the components of the 3D model are ordered in accordance withtheir depth.
 8. The method according to claim 7, wherein a single listof components is generated for all 3D models associated to the 3D videoframe, in which the components of all 3D models are ordered inaccordance with their depth.
 9. An apparatus for transmitting a 3D modelassociated to a 3D video frame, the apparatus comprising: a firstanalyzer for splitting the 3D model into one or more components; asecond analyzer for determining whether a component of the one or morecomponents is hidden by other 3D content; and an output for transmittingthose components of the 3D model that are not hidden by other 3Dcontent, and for subsequently transmitting the remaining components ofthe 3D model.
 10. The apparatus according to claim 9, wherein the other3D content is at least one of 3D video, another component of the 3Dmodel, and another 3D model.
 11. The apparatus according to claim 9,further comprising a list generator for generating a list of componentsof the 3D model, in which the components of the 3D model are ordered inaccordance with their depth.
 12. The apparatus according to claim 11,wherein the list generator is adapted to generate a single list ofcomponents for all 3D models associated to the 3D video frame, in whichthe components of all 3D models are ordered in accordance with theirdepth.
 13. An apparatus for preparing a 3D model associated to a 3Dvideo frame for transmission, the apparatus comprising: a first analyzerfor splitting the 3D model into one or more components; and a secondanalyzer for determining whether a component of the one or morecomponents is hidden by other 3D content.
 14. The apparatus according toclaim 13, wherein the other 3D content is at least one of 3D video,another component of the 3D model, and another 3D model.
 15. Theapparatus according to claim 13, further comprising a list generator forgenerating a list of components of the 3D model, in which the componentsof the 3D model are ordered in accordance with their depth.
 16. Theapparatus according to claim 15, wherein the list generator is adaptedto generate a single list of components for all 3D models associated tothe 3D video frame, in which the components of all 3D models are orderedin accordance with their depth.
 17. A non-transient recording mediumhaving recorded thereon 3D content, wherein the 3D content includes atleast one or more 3D video frames and one or more 3D models associatedto the one or more 3D video frames, wherein components of the one ormore 3D models associated to a 3D video frame are arranged in a list ofcomponents, in which the components of the one or more 3D modelsassociated to the 3D video frame are ordered in accordance with theirdepth.